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Abstract 



Advocates of online learning are optimistic about its potential to promote greater 
access to college by reducing the cost and time of commuting and, in the case of 
asynchronous approaches, by allowing students to study on a schedule that is optimal for 
them. This goal of improved access has been one of the top motivators for postsecondary 
institutions to expand their distance education offerings, which has in turn helped drive a 
strong increase in online course enrollments over the last decade. A series of technology- 
based classroom initiatives has also attracted strong attention from postsecondary 
educators. The enthusiasm surrounding these and other innovative, technology-based 
programs has led educators to ask whether the continuing expansion of online learning 
could be leveraged to increase the academic access, progression, and success of low- 
income and underprepared college students. 

This paper examines the literature for evidence regarding the impact of online 
learning on these populations. First, a research review strongly suggests that online 
coursework — at least as it is currently and typically implemented — may hinder 
progression for low-income and underprepared students. Second, the paper explores why 
students might struggle in these courses, discusses current access barriers to online 
education, and offers suggestions on how public policy and institutional practice could be 
changed to allow online learning to better meet its potential for these students. 
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1. Introduction 



Advocates of online learning are optimistic about its potential to promote greater 
access to college by reducing the cost and time of commuting and, in the case of 
asynchronous approaches, by allowing students to study on a schedule that is optimal for 
them. This goal of improved access has been one of the top motivators for postsecondary 
institutions to expand their distance education offerings (Parsad & Lewis, 2008), which 
has in turn helped drive the strong increase in online course enrollments over the last 
decade (from 10% of enrolled students in 2002 to 29% in 2009; see Allen & Seaman, 
2010 ). 

Meta-analyses comparing academic outcomes between online and face-to-face 
courses have typically indicated that there is no overall difference between the two 
formats but that there is large variation in effects, with some online courses having much 
better outcomes than face-to-face courses and other online courses having much worse 
outcomes (e.g., Bernard et al., 2004; Zhao, Lei, Yan, Lai, & Tan, 2005). A recent meta- 
analysis of the most high-quality studies (U.S. Department of Education, 2009) suggested 
that online learning results in similar or better outcomes than does face-to-face learning, 
although the authors cautioned that the positive effect for online learning outcomes was 
stronger when contrasting hybrid-online courses to face-to-face courses than when 
contrasting fully-online courses to face-to-face courses. In addition, they noted that the 
positive effect was stronger when the hybrid-online course incorporated additional 
materials or time on task, which were not included in the face-to-face course. 

At the same time, a series of technology-based classroom initiatives — such as 
Western Governors University, Carnegie Mellon’s Open Learning Initiative, and the 
National Center for Academic Transformation’s course redesign process — has attracted 
strong attention from postsecondary educators. The enthusiasm surrounding these and 
other innovative, technology-based programs has led educators to ask whether the 
continuing expansion of online learning could be leveraged to increase the academic 
access, progression, and success of low-income and underprepared college students. To 
provide an evidence-based perspective on these questions, the current review examines 
the literature for information regarding the impact of online learning on these 
populations. 
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In academic and policy discussions, sometimes little distinction is made between 
fully online courses, partially online courses, and courses that are essentially face-to-face 
but which incorporate some resources online; yet each of these types of courses may have 
very different implications for access, progression, and student learning. Following Allen 
and Seaman (2010), this paper uses the term “online” for fully online courses (80% or 
more of the course is conducted online) and “hybrid” for partially online courses (30%- 
79% of the course is conducted online). As noted above, online learning is thought to 
have strong potential to increase student access. In contrast, many hybrid courses require 
students to attend a substantial proportion of time on campus; 1 thus, scaling up hybrid 
course offerings is unlikely to improve access for low-income students who have work, 
family, or transportation barriers to attending class in a physical classroom at specified 
times. Perhaps in part due to community colleges’ strong emphasis on access, online 
courses are much more prevalent than hybrid courses in the community college setting: 
75% of community colleges offer online courses, while only 15% offer hybrid courses 
(Instructional Technology Council, 2010). Institutions providing hybrid courses may also 
offer fewer sections of hybrid than online courses; for example, in a recent study of 
Virginia’s 23 community colleges, first-year students took 9% of their courses online but 
only 3% of their courses via a hybrid mode (Jaggars & Xu, 2010). 

Given the much higher prevalence of online versus hybrid learning in community 
colleges, combined with online education’s strong promise of improved student access, 
this paper focuses on online courses. I begin with a research review that strongly suggests 
that online coursework — at least as currently and typically implemented — may hinder 
progression for low-income and underprepared students. I then explore why students 
might struggle in these courses, discuss current access barriers to online education, and 
offer suggestions on how public policy and institutional practice could be changed to 
allow online learning to better meet its potential for these students. 



1 For example, Jaggars and Bailey (2010) note that of the 23 hybrid courses examined in a U.S. Department 
of Education (2009) meta-analysis, 20 required the students to physically attend class for the same amount 
of time that students in a face-to-face course would attend; the online portions of these courses were either 
in on-campus computer labs or were supplemental to regular classroom time. 
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2. Review of the Research 



This review of the literature focuses on the extent to which low-income and 
underprepared students may have a more successful college experience by taking 
advantage of online courses. As little research focuses explicitly on this target population, 
the review includes studies on all postsecondary students, including graduate students, 
with an eye to how the overall postsecondary results may or may not be generalized to 
low-income and underprepared students. The postsecondary inclusion criterion 
distinguishes this review from other recent analyses of the online learning literature, 
which each included studies from a mixed variety of settings, including K-12, college, 
and work-based employee training contexts (Bernard et al., 2004; Zhao, Li, Yan, Lai, & 
Tan, 2005; U.S. Department of Education, 2009). 

This review includes studies that compared online (80% or more of the course 
conducted online) and face-to-face courses (less than 30% of the course conducted 
online) in terms of students’ course enrollment, completion, performance, or subsequent 
academic outcomes. Student satisfaction was not included as one of the outcomes, largely 
because the relationship between student satisfaction with a particular course and 
subsequent academic success is not clear. Given the larger cultural and economic sea 
change surrounding Internet usage around the turn of the 21st century, studies published 
prior to 2000 may not be relevant to today’s online context; thus, studies published prior 
to 2000 were discarded. Also discarded were studies that focused on short educational 
interventions (e.g., Beyea, Wong, Bromwich, Weston, & Fung, 2008, who tested a 15- 
minute intervention; or Erickson, Chang, Johnson, & Gruppen, 2003, who tested a one- 
hour intervention). Such short interventions cannot address the challenging issues 
inherent in maintaining student attention, learning, motivation, and persistence over a 
course of several months. Also discarded were studies that allowed students to self-select 
into either online or face-to-face courses without attempting to control for any potential 
differences between the student groups. Finally, this paper considers only studies 
conducted in the United States and Canada. 2 Overall, this paper looks at 34 papers (some 
including multiple studies, resulting in a total of 36 studies) that met the review criteria. 

2 While the scope of the paper initially also included English-speaking countries with similar demographic 
and educational contexts (e.g., the United Kingdom, Ireland, and Australia), this did not yield any 
additional studies that matched the remaining criteria. 
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In the discussion below, studies are grouped according to the outcomes examined: access 
(increasing enrollments that would not otherwise occur), completion (finishing the 
course), learning outcomes (typically the final grade or exam score in the course), and 
progression (movement into or performance in subsequent courses and semesters). 

As a preview to the summary of the research literature, I note at the outset that: 

• Few studies focused explicitly on the potential for online 
learning to support the success of low-income or academically- 
underprepared students; 

• No studies explicitly addressed the extent to which online 
learning options increase college enrollment rates in 
comparison to face-to-face learning options; 

• Fewer than half of the studies compared course completion 
rates between the online and face-to-face format; 

• Most studies compared student learning outcomes, such as 
assignment grades, final exams, and final course grades, 
between the online and face-to-face settings; and 

• Only two studies compared online and face-to-face courses in 
terms of outcomes subsequent to the online course. 

The 36 studies included in this paper are summarized in Table 1 (Appendix). 

2.1 Courses Studied 

This section begins with a brief description of the types of online courses studied. 
Of the 36 studies, 21 (58%) focused on university courses, 10 (32%) on community 
college courses, four (1 1%) on graduate-level courses, and one on a course in a for-profit 
institution. Approximately one third focused on math, statistics, computer science, and 
other science-oriented courses; another third on professionally-oriented areas such as 
health, law, business, and education; and the final third on liberal arts areas including 
economics, English, and critical reasoning. A few studies examined courses across an 
entire institution and thus provided no relevant information on specific course subjects or 
instructional characteristics. Many studies of specific courses also did not provide full 
information regarding the features of the online course. However, even the partial 
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information included in each study demonstrates that the courses varied widely in their 
instructional features (see Table 1, column 4). 

Instructional delivery method. Of the 19 studies that specified the method by 
which instructional material was delivered to students, 10 (53%) used a text-based or 
slideshow-based presentation, which was sometimes supplemented with short video or 
audio clips, or with self-test exercises. Video or audio of the instructor’s full lecture was 
the next most popular instructional method, used in five (26%) of the studies. Three 
studies (16%) specified using tutorial software programs to deliver content, along with a 
fourth study that offered students both lecture notes and interactive tutorials. 

Student interaction. It was difficult to determine the degree to which student- 
student and student-instructor interactivity tools were incorporated into courses, as most 
studies did not mention this component. Six studies noted that asynchronous discussion 
or synchronous chat was available to students, but these studies either specified that 
students were not required to participate or did not specify whether participation was 
required as part of the course grade. Eight other courses required participation; of these, 
six used asynchronous discussion and two used synchronous chat sessions. Finally, one 
paper studied an online course that incorporated interaction via face-to-face discussion 
sessions (Schemes, Leinhardt, Smith, & Cho, 2005). This study is of particular interest, 
as the course was developed by the Carnegie Mellon Open Learning Initiative, which is 

o 

often cited as an exemplary online course development program. The Scheines et al. 
(2005) paper includes five separate studies. In the first two studies, face-to-face 
discussion attendance was not required, and student attendance at these sessions was 
quite low: in study 1, online students attended an average of 20% of the sessions 
(compared to 85% attendance by face-to-face students), and in study 2, attendance was 
less than 9% for online students (85% for face-to-face students). Given that this version 
of the online course did not require face-to-face discussions, the first two studies met this 
review’s inclusion criterion in terms of examining a “fully online” course. In the 
subsequent studies in the paper, however, discussion attendance was required, and online 



3 Other studies conducted by the Open Learning Initiative are not included, as they concern hybrid courses 
(see Lovett, Meyer, & Thille, 2008, in which students were required to meet with an insUuctor at least once 
a week) or examine online courses without comparing them to face-to-face courses. 
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student attendance at these sessions rose to 71%. I considered these courses to be hybrid 
courses and excluded them from this review. 

Student supports. Nine studies noted that the course under examination required 
an initial face-to-face orientation session for online students; these orientations were 
viewed as helpful in assisting students to understand the academic demands of the course, 
as well as the technical features of the web-based interface. One study noted that a face- 
to-face orientation was available but that some students did not take advantage of it, and 
recommended that the orientation be mandatory in the future (Rosenfeld, 2005). Only 
three studies explicitly mentioned the provision of other non-instructional supports (e.g., 
technical support, library service support) for students in the online course. Navarro and 
Shoemaker (2000) noted that technical support was available by email within 24 hours; 
Rosenfeld (2005) studied a community college that offered free Internet access to 
students enrolled in online courses and provided technical support 14 hours per day, 6 
days per week (though the method of support provision was unclear); and Vroeginday 
(2005) noted that library resources and technical support were integrated into the online 
course platform. 

Typical online courses. It is not clear the extent to which the courses examined 
in these studies are representative of the “typical” online course, as there is no 
comprehensive survey of online course features in postsecondary education. Of the 
studies of specific courses included in this review, most were conducted by the instructor 
or by departmental colleagues. It seems likely that these courses were selected for study 
because they were thought to be high-quality examples of online learning. In contrast, 
institution-wide or system-wide studies (e.g., Rosenfeld, 2005; Jaggars & Xu, 2010) 
should capture the effects of typical online courses in that institution or system. 

Having provided an overview of the online courses included in this review, I 
move on to compare student outcomes between online and face-to-face courses. This 
section begins with studies that examine course completion rates, followed by those that 
examine learning outcomes, and then those that examine student progression. Given that I 
found no studies that examined student access, the paper postpones further discussion of 
that issue until a subsequent section of the paper. 
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2.2 Course Completion 

Course completion is a fundamental measure of student success, as many students 
are lost from the postsecondary pipeline when they attempt but fail to complete courses. 
Underprepared students who withdraw from a course mid-semester run the very real risk 
of never returning to successfully complete the course, thereby prohibiting progression to 
the next course in the sequence (see, e.g., Bailey, Jeong, & Cho, 2010). Since 
underprepared students must complete developmental courses before progressing into 
courses at the college level, low completion rates in remedial courses are especially 
problematic. 

Online courses, like other forms of distance education, clearly have higher mid- 
semester withdrawal rates than do face-to-face courses (Beatty-Guenter, 2003; Carr, 2000; 
Chambers, 2002; Moore, Bartkovich, Fetzner, & Ison, 2003). This difference may be 
particularly pronounced among underprepared students. For example, one study of 
community college students in developmental mathematics observed that 73% of face-to- 
face students completed the course with a grade of A, B, or C, while only 51% of online 
students did so (Summerlin, 2003). However, some practitioners and researchers argue 
that high online withdrawal rates are due not to the course format, but to the 
characteristics of online students (Howell, Laws, & Lindsay, 2004; Hyllegard, Heping, & 
Hunter, 2008). Thus, for this review of the literature, I looked particularly for studies that 
compared online and face-to-face completion rates while controlling for student 
characteristics. Ten of the studies in this review did so, and their results fell into two sets: 
(1) Four studies found no significant difference in withdrawal rates and (2) six studies 
found higher withdrawal rates for online students. 

Studies showing no difference in course withdrawal rates. One randomized 
and three controlled studies showed no difference between online and face-to-face 
student completion rates. Waschull (2001, study 2) examined an introductory psychology 
course at a non-selective technical college. Forty-one students enrolled in one of two 
sections of a course taught by Waschull; she then randomly selected one section to be 
conducted online. While participants in this study were not strictly randomized, it is 
reasonable to assume that student selection of a given course was independent of the 
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assignment of sections to the online condition. Attrition was fairly low (11% in online, 

9% in face-to-face) and did not significantly differ between the two courses. 

The remaining three studies were conducted by Dutton and colleagues (J. Dutton 
& M. Dutton, 2005; J. Dutton, M. Dutton, & Perry, 2001, 2002), who examined a C++ 
programming course and a business statistics course at a selective university. These 
studies are problematic, because the analyses of completion included controls for student 
behaviors that occurred after the online course began; thus, these behaviors could have 
been caused, at least in part, by the online course mode itself. In all three Dutton studies, 
raw data indicated that the online group had substantially higher withdrawal rates (e.g., in 
the 2001 study, 21% online versus 6% face-to-face). However, after controlling for 
homework grades, these differences disappeared. The authors argued that the homework 
measure captured preexisting student motivation, given that students were allowed to re- 
do their homework several times, such that a motivated student could receive a perfect 
score on every assignment. However, it is possible that the online course format 
adversely affected student motivation and engagement, which in turn led to both poor 
homework grades and eventual student withdrawal. Thus, one valid interpretation of 
these studies’ results might be that the online format of the course indeed contributed to 
higher withdrawal rates. 

Studies showing higher course withdrawal rates for online students. All six of 

the controlled studies showing higher withdrawal rates for online courses dealt with 
community college students. These studies are especially relevant to this paper, since 
community colleges disproportionately serve low-income and academically- 
underprepared students (Adelman, 2005). Four studies dealt explicitly with 
developmental students, and the remaining two dealt with students in introductory 
“gatekeeper” college-level courses (such as first-year English). 

The six studies controlled for a wide array of variables, including psychological 
characteristics such as locus of control, math anxiety, learning style, or reasons for 
choosing the course (Blackner, 2000; Zavarella, 2008); prior achievement, including 
academic placement test scores, credits accrued, or GPA (Carpenter, Brown, & Hickman, 
2004; Jaggars & Xu, 2010; Rosenfeld, 2005; Xu & Jaggars, 2010; Zavarella, 2008); basic 
demographics such as gender, ethnicity, age, or marital status (Carpenter et al., 2004; 
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Jaggars & Xu, 2010; Rosenfeld, 2005; Xu & Jaggars, 2010; Zavarella, 2008); federal 
financial aid eligibility (Jaggars & Xu, 2010; Xu & Jaggars, 2010); semester-specific 
variables such as credit load and timing of registration (Carpenter et al., 2004; Jaggars & 
Xu, 2010; Xu & Jaggars, 2010); and instructor characteristics or effects (Carpenter et al., 
2004; Jaggars & Xu, 2010). Across the studies, online students tended to have estimated 
course withdrawal rates that were 10 to 15 percentage points higher than those of face-to- 
face students. 

Overall, the 10 studies examining student course completion suggest that students 
are less likely to complete courses if they take them online, although this tendency may 
be particularly pronounced among community college students, who tend to be 
disproportionately low-income and academically underprepared. 

2.3 Student Learning Outcomes 

The paper next turns to the literature on learning outcomes to assess whether 
students who complete online courses have equally successful academic outcomes in 
comparison to those who complete face-to-face courses. As a general experimental 
research principle, when comparing outcomes between two groups who persist to the end 
of a study, it is particularly important to examine whether the two groups had relatively 
equal levels of attrition. To illustrate the importance of this principle, consider two 
courses, Course 1 and Course 2, each with similar students at the outset (Figure 1). 
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Figure 1 

An Illustration of Unequal Withdrawal Rates 
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At the outset of the semester, each course has a similar mix of students: each has 
four students who are typically “A students,” six that are typically “B students,” and so 
on. Over the course of the semester, some students with poorer academic preparation 
withdraw from each course. However, something is different about Course 2, which 
causes twice as many poor-performing students to withdraw. The remaining students 
perform at their usual level of academic success in each course. If the two courses are 
then compared in terms of the academic performance of the completers , Course 2 will 
appear to be the superior course. 

Given the importance of establishing whether two groups have equal rates of 
withdrawal before comparing their outcomes, together with the fact that online courses 
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typically have higher withdrawal rates, it is curious that of the 33 studies that examined 
learning outcomes, only 18 even mentioned course completion rates. This omission is 
particularly perplexing given that all articles were published after a much-cited review of 
online learning effectiveness (Phipps & Merisotis, 1999), which criticized the literature 
for failing to adequately address the issue of withdrawal rates. Of the 18 learning 
outcomes studies that mentioned withdrawal rates, nine were discussed in the section 
above; the remaining nine compared learning outcomes while ignoring completion. Six of 
these nine dealt with fairly well-prepared students and appropriately noted that overall 
course withdrawal rates were negligibly low or equivalent between groups (Caldwell, 
2006; Figlio, Rush, & Yin, 2010; Partrich, 2003; Schemes, Leinhardt, Smith, & Cho, 
2005, studies 1 and 2; Y.-C. Wang, 2004); two noted that online dropout rates were 
significantly higher (Carey, 2001; Summerlin, 2003); and one did not explicitly test what 
appeared to be substantially higher withdrawal rates among online students (Piccoli, 
Ahmad, & Ives, 2001). 4 

In terms of other methodological characteristics of the 33 learning outcomes 
studies, 8 were randomized studies, and 5 more were relatively well controlled, but the 
remainder raised methodological concerns about the handling of control variables. The 
paper discusses each of these three sets of studies in more detail below. 

Randomized studies of learning outcomes. I found eight studies of learning 
outcomes that randomized students into online versus face-to-face conditions. In general, 
these studies were reasonably well designed with fairly equivalent student groups and 
similar curricula between the online and face-to-face sections of each course. Five 
showed no significant difference between groups (Caldwell, 2006; Piccoli, Ahmad, & 
Ives, 2001; Schemes, Leinhardt, Smith, & Cho, 2005, studies 1 and 2; Waschull, 2001, 
study 2). One showed no overall differences between groups but noted that among 
Hispanics, males, and students with low prior GPAs, students in the online condition 
scored significantly lower on in-class exams (Figlio, Rush, & Lin, 2010). Another study 

4 In Piccoli et al. (2001), from the course’s first week to the final, online retention rates were 74% and face- 
to-face were 94%; a comparison of these proportions yields a highly significant difference, % = 13.15, p < 
.001. The authors noted that online students became significantly less satisfied with the course when they 
encountered more difficult material in the second half of the semester while face-to-face students did not. 
Dropout rates from the midterm to the final reflect this dissatisfaction: online retention rates during this 
period were 86% and face-to-face rates were 96%, % 2 = 5.43, p < .05. 
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showed a negative effect for online learning, primarily because online students were less 
likely to complete their homework (Mentzer, Cryan, & Teclehaimanot, 2007). Finally, 
one study showed a positive effect for online learning, which the authors admitted might 
be due to the small class size for online students, in comparison to the very large lecture- 
hall class for face-to-face students (Poirier & Feldman, 2004). 

Overall, then, the eight randomized studies suggest no strong positive or negative 
effect of online learning in terms of exam or course grades. These results, however, are 
not readily generalized to low-income and underprepared students: most of the studies 
had small sample sizes, were conducted at selective or highly- selective universities, and 
either had negligible student withdrawal or did not mention withdrawal at all. Only Figlio 
et al. (2010) explicitly examined impacts among less-prepared students, finding that such 
students perform significantly more poorly in online courses. 

Well-controlled studies of learning outcomes. Of the 25 studies of learning 
outcomes that attempted to adjust for student self-selection into online courses using 
control variables, five were relatively well controlled, while the remaining 20 manifested 
methodological limitations with regard to their inclusion of controls. Of the five 
relatively well-controlled studies, three showed negative effects for online learning, one 
showed positive effects, and one showed no significant difference. Two of the five 
studies focused on economics students at selective universities, while three focused on 
community college students in developmental or gatekeeper courses. Each of these 
studies provides interesting insights when explored in more detail. 

Studies of university economics courses. Brown and Liedholm (2002) examined 
online (N = 89) and face-to-face (N = 363) sections of a microeconomics course, using a 
regression controlling for gender, ethnicity, previous math coursetaking, ACT score, prior 
GPA, and previous credits accrued. Students in the online courses were significantly 
better prepared at the outset, with significantly higher ACT scores and accrued college 
credits; after controlling for these factors, however, students in the online course 
performed more poorly than those in the face-to-face course. The authors extended their 
regression model to predict online student outcomes if online students had instead chosen 
to enroll in a face-to-face course and note that online students’ course grades “would 
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have risen by a significant 5.79 percentage points” if they had taken the course face-to- 
face. 

A study of online (N = 59) and face-to-face (N = 67) economics students across 
three universities (Coates, Humphreys, Kane, & Vachris, 2004) administered a common 
post-test to students at the end of the semester. The model comparing post-test scores 
between groups included an impressive array of covariates, including SAT, hours 
employed, commuting time to campus, financial aid status, age, an economics pre-test, 
and instructor effects. In addition to traditional regression, the authors also used more 
advanced techniques (2SLS and endogenous switching) to control for student self- 
selection effects. Similar to Brown and Liedholm (2002), they concluded that online 
students performed more poorly than face-to-face students. For example, a typical 
economics student who took the course online rather than face-to-face would score 18 
percentage points lower on the final; moreover, underclassmen seemed particularly 
vulnerable to this effect, leading the authors to note that teaching economics courses 
online in community colleges is probably not a good policy. Contrary to Brown and 
Liedholm, however, their endogenous switching model suggested that students who self- 
selected into the online class performed better than students with similar characteristics 
who chose the face-to-face class. Thus, while the average student performed more poorly 
in an online class, the type of student who was most likely to choose an online class 
performed better in that setting. 

Neither study mentioned student withdrawal, and in general the students were 
well prepared. As Coates et al. (2004) noted of their sample: “The typical student carried 
a 2.8 GPA on a 4 point scale, scored about 1 100 on the SAT and considered himself to be 
good at math” (p. 537). Both studies reach the same general conclusion that the typical 
student does more poorly in online courses. However, it is difficult to judge the extent to 
which these studies apply to low-income and underprepared students. 

Studies of community college students. As noted in the discussion of course 
completion in an earlier section, Carpenter et al. (2004) controlled for a variety of factors 
and found that developmental writing students were significantly more likely to withdraw 
from an online course than from a face-to-face course. However, students who completed 
the course were more likely to earn a good grade (2.5 or higher) in the course, net of 
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controls. The authors acknowledge that this effect could be due to the substantially higher 
withdrawal rates in the online sections. To further explore this possibility, they examined 
the reading and writing placement test scores of students who withdrew from each 
section and found that students with lower placement scores were more likely to 
withdraw from the online section, while students with higher scores were more likely to 
withdraw from the face-to-face section, leaving the online section with students who 
were better prepared at the outset. This pattern gives weight to the notion, discussed at 
the beginning of this section, that differential withdrawal rates can result in misleading 
comparisons between students who complete online and face-to-face courses. 

Summerlin (2003) focused on a developmental mathematics course, and 
compared a sample of online students (N = 79) to a randomly-drawn sample of face-to- 
face students (N = 143) in terms of their end-of-semester scores on a state mathematics 
exam. Across the college, observed withdrawal from the online sections was substantially 
higher; but among those students in the author’s subsample who completed the course, 
exam scores were similar between the groups after controlling for reading ability, age, 
gender, and ethnicity. 

Finally, Xu and Jaggars (2010) examined students across 23 Virginia community 
colleges who took the system’s introductory college-level English course either online (N 
= 1,052) or face-to-face (N = 12,921). As noted in the completion section above, that 
study found that online students were much more likely to withdraw from the course. In 
order to eliminate the potential confounding effect of unequal withdrawal in the analysis 
of learning outcomes, the authors used a propensity score method that matched each 
student who completed the English course online with an otherwise extremely- similar 
student who completed the English course face-to-face. This analysis represents the only 
study in this review that explicitly addressed and removed the potential influence of 
unequal attrition, and it found that online students who completed the course were 
significantly less likely to earn a good grade (C or above) than were face-to-face students. 

Less well-controlled studies. Students who choose online coursework are often 
quite different from those who choose face-to-face courses. In conjunction with a 
comparison of course completion or learning outcomes, several studies included in this 
review compared demographic or academic characteristics between online and face-to- 
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face students and found distinct patterns of differences. For example, the study of 
Virginia community college students (Jaggars & Xu, 2010) found that students who 
chose to take online courses were significantly more likely to be female. White, English- 
fluent, independent of their parents, and above 25 years old; they were also more likely to 
have applied for financial aid, been involved in a college-credit dual-enrollment program 
in high school, and been academically prepared at college entry. Similarly, Carpenter et 
al.’s (2004) study of developmental community college students found that online course 
takers were more likely to be female. White, older, part-time students, and to have higher 
reading and writing placement exam scores. Differences between online and face-to-face 
students are also apparent in the university environment: in Brown and Liedholm (2002), 
online students were significantly less likely to be Black or student athletes, and had 
significantly higher ACT scores and accrued credits; in Coates et al. (2004), online 
students were significantly less likely to be underclassmen, traditional- age, non- 
employed, living on campus, or receiving financial aid. 

Among the studies of learning outcomes, 20 were classified as “less well 
controlled.” These studies typically included only one or two controls — often without any 
reflecting the distinct characteristics of online students discussed above — raising the 
strong possibility that their results are driven merely by pre-existing differences between 
online and face-to-face students. Approximately half used either a pre-test 5 or a learning 
styles assessment as their primary control, but typically failed to mention whether the 
measure was a useful control. For example, Schoenfeld-Tacher, McConnell, and Graham 
(2001) compared final exam scores between online and face-to-face students in an upper- 
division tissue biology course at a state university, controlling for pretest scores. Yet the 
pre- and post-test scores were not related (with rf = .000). Accordingly, including only 
this variable in the model was equivalent to including no control at all. Similarly, studies 
that included learning or cognitive style as a control typically found little or no 
relationship between styles and learning outcomes. 

Overall, across the 20 studies in this set, nine used only a single control; three 
collected an array of student information but analyzed each characteristic in a separate 
analysis, an approach that is insufficient to ensure comparability between groups on the 

5 Or, similarly, compared groups in terms of a gain score between pre- and post-test. 
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set of observed characteristics (Navarro & Shoemaker, 2000; Sankaran & Bui, 2001; 
Vroeginday, 2005); and five used only two or three controls, with little information as to 
the efficacy of the controls in reducing self- selection bias. Finally, as noted in a previous 
section, three papers by Dutton and colleagues (J. Dutton & M. Dutton, 2005; J. Dutton, 
M. Dutton, & Perry, 2001, 2002) committed the error of controlling for student behaviors 
that arose subsequent to enrollment in the online course. 

In terms of results, several studies included multiple outcomes that showed mixed 
results, and some studies’ statistical results were unclearly reported. Taking the broad 
view across studies, however, most showed either positive effects or no effects for online 
learning. Yet given the methodological weaknesses of the studies, it is difficult to 
interpret these results. Relatively poor controls raise the question of whether results were 
driven in part by student self- selection, and the fact that few studies examined withdrawal 
raises the question of whether results are driven in part by differential attrition. Moreover, 
almost all the studies were conducted with university or graduate-level students, which 
makes unclear their relevance to low-income or underprepared students. 

2.4 Student Progression 

For low-income and underprepared college entrants, retention into subsequent 
semesters, as well as progression to the next course in the program sequence, are critical 
issues. As several other papers in this series point out, many underprepared students do 
not complete their initial developmental education courses. Those who fail to exit the 
developmental sequence and progress to introductory college-level “gatekeeper” courses 
are unlikely to successfully transfer or earn a credential (Bailey, Jaggars, & Cho, 2010). 
Given the importance of progression, it is surprising that only two of 28 studies explored 
whether students who complete an online course are on equal footing with their 
classroom-based equivalents in terms of subsequent success. The first study (Summerlin, 
2003), discussed in more detail in the learning outcomes section, noted that students 
taking developmental math online were less likely to subsequently pass college-level 
algebra, but the study provides no further controls in that analysis. The second study was 
the analysis of 23 Virginia community colleges (Jaggars & Xu, 2010), which controlled 
for a wide array of student characteristics and found that: 
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• Students who took online coursework in early semesters were 
less likely to return to school in subsequent semesters. For 
example, among the 2004 cohort, those who took one or more 
online courses in the first fall semester were significantly less 
likely to return in the spring, with adjusted retention rates 5 
percentage points lower than those of students who took a fully 
face-to-face curriculum (69% vs. 74%). 

• Students who took developmental math and English courses 
online were much less likely to subsequently succeed in 
college-level math and English. For example, adjusted 
enrollment rates into college-level English among the 2004 
cohort were almost 30 percentage points lower among those 
who took their developmental English course online compared 
to those who took their developmental course face-to-face. 

Among those who moved on to college-level English, those 
who took the developmental course online had success rates 9 
percentage points lower than those who took the developmental 
course face-to-face. 

• Students who took a higher proportion of credits online were 
less likely to attain a credential or to transfer to a four-year 
institution. For example, in the 2004 cohort, those who took 
only a few credits online had award/transfer rates that were six 
percentage points higher than those who took a substantial 
number of credits online. 

While this evidence is limited to community colleges in one state, it does suggest 
that online learning may undercut student progression among underprepared students 
who may be struggling with learning difficulties or other problems that interfere with 
student success. 

2.5 Summary of the Research 

Overall, it seems that community college students who take online courses are 
more likely to withdraw from them and that this tendency toward withdrawal is not due 
to the measured characteristics of those students. Moreover, tentative evidence suggests 
that taking online courses may discourage students from returning in subsequent 
semesters and moving on to subsequent courses in their program sequence. 
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The findings in terms of learning outcomes among students who complete online 
and face-to-face courses are more equivocal. Randomized studies of student learning 
outcomes (primarily conducted at selective institutions with low course withdrawal rates) 
suggested that students who complete online and face-to-face courses have similar end- 
of-semester learning outcomes. Studies that allowed for self- selection between delivery 
methods and included few controls in the analysis (many of which were performed with 
university students) tended to find no effect or positive effects for online learning. Other 
studies with stronger statistical methodologies (more than half of which were conducted 
with community college students) tended to find negative effects for online learning. 

The variation in results across the three sets of learning outcomes studies could 
spring from two different causes. First, online learning may be more effective among 
university students than among community college students. Second, when considering 
the non-randomized studies, the discrepancy between the results of less-controlled and 
more-controlled analyses is not necessarily surprising, as several studies suggested that 
students who select online learning tend to have superior academic characteristics. Thus, 
studies that do not control for these characteristics will tend to find that online students 
perform better. Moreover, if weak students are more likely to withdraw from online 
courses, then the students who remain in the online course may outperform those in the 
face-to-face section simply on that basis. 

Taking all of these points into account, it is difficult to judge whether students 
who complete online courses perform similarly to those who complete the equivalent 
course face-to-face. The evidence suggests, however, that colleges that are focused on 
improving student success should proceed cautiously in expanding online course 
offerings. Until more evidence is gathered, the results of Coates et al. (2004) suggest that 
online coursework may not be optimal for less skilled students, but may instead be most 
useful to well-prepared students who need the flexibility of an online course to complete 
their program of study. 
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3. Why Are Online Course Completion Rates Lower? 

The body of research evidence discussed above suggests that lower online 
completion rates are not simply due to the characteristics of students who choose to enroll 
in those courses. That is, independent of students’ prior skills, the online format of the 
course itself might pose difficulties for students. What, then, might be the characteristics 
of the online course itself that would induce students to withdraw? The theoretical and 
research literature suggest at least three possibilities: technical difficulties, increased 
“social distance,” and a relative lack of structure inherent in online courses. 

3.1 Technical Difficulties 

Technical difficulties can be particularly frustrating for students who are working 
under time pressure to turn in homework, complete quizzes and exams, participate in 
discussion sessions, take notes from streaming video lectures, or participate in a myriad 
of required web-based activities. When researchers talk to students about their online 
course experience, technical problems (both unavoidable system-based problems and 
difficulties caused by the user’s lack of familiarity with the system) surface as a frequent 
complaint (e.g., Bambara, Harbour, Davies, & Athey, 2009; El Mansour & Mupinga, 
2007; Hara & Kling, 1999; Mupinga, Nora, & Yaw, 2006; Navarro & Shoemaker, 2000; 
Rivera, McAlister, & Rice, 2002; L. Wang, 2008). For example, in one study (Zavarella, 
2008) the predominant reasons for student withdrawal from online courses were technical 
problems, computer-based learning issues, or other factors explicitly related to the online 
nature of the course. 

3.2 Social Distance 

As Karp (201 1) discusses in her paper in this series, numerous educational 
theorists argue that in order to improve retention rates, institutions need to build students’ 
sense of belonging and commitment; this might be accomplished by encouraging social 
relationships through communities of learning and by otherwise helping students feel that 
the institution “cares.” Similarly, distance education theorists argue that in order to 
engage and motivate students, distance courses must explicitly build social or emotional 
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connections between students and instructors, as well as among students (e.g., Fontaine, 
2010; Garrison, Anderson, & Archer, 2003). And indeed, a sense of “social presence,” or 
the projection of an authentic persona that connects to others involved in the course, 
correlates strongly with online student course satisfaction, performance, and retention 
(Boston et al., 2009; Gunawardena & Zittle, 1997; S. Y. Liu, Gomez, & Yen, 2009; 
Picciano, 2002; Richardson & Swan, 2003). 

Studies of students’ experiences, however, suggest that many online courses lack 
a sense of social presence. A study of online courses in the Virginia community college 
system noted that 43% of students complained of inadequate levels of feedback and 
interaction in their online courses (Virginia Community College System, 2001). A 
qualitative study of online university students reported that many felt “lost in 
cyberspace,” without a personal connection to the instructor or other students (El 
Mansour & Mupinga, 2007, p. 247). Similarly, in a qualitative study of online students 
enrolled in four high-risk community college courses (HRCs), students’ primary 
complaint was a sense of isolation in their online course (Bambara et al., 2009). As the 
authors noted in terms of student-instructor interaction: 

HRC participants did not see anyone, and they did not have 
a sense that anyone was present. David explained, “I don’t 
feel like there was an instructor presence. ... I don’t feel 
like there was anything that I was learning from the 
instructor. The instructor was simply there as a Web 
administrator or as a grader.” (pp. 224-225) 

Students also felt a lack of interaction with one another: 

What our participants found was aptly described by 
Samantha as follows: “No interaction between the students, 
student interaction is nonexistent! I know nothing about 
these people!” They had no sense of community within the 
HRCs, no peer interaction. Geraldine remarked, “I was just 
sort of on this island, all by myself.” David echoed her 
sentiments when he said, “I felt like, specifically in that 
[HRC] class, I was alone and adrift.” (p. 225) 

The lack of social presence and interaction in some online courses may be due to 
a lack of instructor time and training. In a study of audio/video technology, many 
instructors seemed unaware of how these technologies could be effective tools for student 
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collaboration (S. Liu, 2005). In another study of teachers’ online discussion facilitation 
(Zhai & Liu, 2005), approximately half of the instructors rarely moderated the discussion, 
while only a few participated extensively through “questioning, prompting responses, 
probing discussion with more questions, redirecting the flow of discussion, recognizing 
good posts, summarizing discussion, and modeling social presence” (p. 2). Instructors 
said that several issues affected their degree of presence: not knowing effective 
moderation strategies, having too little time, and worrying about being too authoritarian. 
Dillon and Greene (2003) note that teachers are correct to worry about unintentional 
messaging when they interact with students online: 

Online communication can be easily misinterpreted, due in 
part to the lack of visual and facial cues. Online teachers 
are encouraged to provide timely and detailed feedback. 

However ... they often do not have any information about 
how the student responds to this feedback. In fact, students 
may misinterpret a high level of feedback as negative 
feedback when in reality a teacher is merely posing 
questions to stimulate student thinking, (p. 241) 

Savvy teachers are all too aware of the possibility of online miscommunication. 
And unless they are explicitly trained in how to manage positive online communications 
in a time-effective way, they may default to a psychologically safe and time-saving 
position of minimal presence. 

3.3 Lack of Structure 

Beginning with correspondence courses, distance learning has traditionally been a 
less-structured alternative to the face-to-face classroom. Although the typical 
postsecondary online course has specific assignment due dates and exam dates, it remains 
less structured than the typical on-campus class. Most online courses are asynchronous 
(Parsad & Lewis, 2008), allowing students to access course materials and work on 
assignments at times that are most convenient to them, and to use idiosyncratic strategies 
to access, navigate, digest, and apply the material. While this flexibility is a boon to some 
students, there is also suggestive evidence that the relative lack of structure in online 
courses leads some to procrastinate or fall behind on assignments. In the studies reviewed 
in the first section of this paper, online students were less likely to complete their 
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homework (Mentzer et al., 2007) and had lower participation grades (Grayson, 
MacDonald, & Saindon, 2001). Two studies also noted that online students were more 
likely to turn in assignments late with the excuse of (often unverifiable) technical 
difficulties (Dutton et al., 2001; Ferguson & Tryjankowski, 2009). Another study 
commented that several online students expressed dissatisfaction with the course because 
they felt that they had to teach the material to themselves; as the authors put it, students 
struggled with the “high degree of learner control” (Piccoli et al., 2001). 

In a recent study of students’ perceptions of important instructor behaviors in 
online courses (Sheridan & Kelly, 2010), the most important element of online instructor 
behavior was the clear communication of expectations, requirements, due dates, and 
course activity instructions. Such teacher behavior may benefit students by providing a 
strong structure of timeframes and explicit instructions that help keep students on track 
across the semester. 

Distance education theorists argue that a low-structure approach is more 
appropriate for some learners than others. Moore and Kearsley (2004) note that some 
students have a strong capacity to structure their own learning and thrive when they are 
able to take control, while others need much more guidance in order to succeed. For 
example, perhaps the university economics students in Coates et al. (2004) who explicitly 
chose online learning were making an appropriate choice because they thrived in a low- 
structure setting, while in contrast, the typical student was not prepared to succeed in that 
setting. 

The observation that some students perform poorly without a strongly structured 
environment does not imply that students should never have to learn to succeed in a less- 
structured context. On the contrary, as Dillon and Greene (2003) argue, 

The most important goal of all education, including 
distance education, is to help learners learn how to leam ... 
in a variety of situations and under a variety of conditions, 
because that is the nature of the learning society in which 
we live. (p. 238) 

However, rather than expecting that students will automatically adapt to a new and 
perhaps confusing low- structure online environment, online instructors may need to 
scaffold students to succeed in that environment by providing clear expectations and 
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instructions, as well as by using activities and processes that explicitly teach 
metacognition, system knowledge, and self-efficacy (Hannafin, Hill, Oliver, Glazer, & 
Sharma, 2003). 

3.4 Lack of Student Supports 

As the foregoing discussion suggests, most online students are not getting the 
supports they need to help them succeed in the online context. Mixed-offering institutions 
(those who provide both on-campus and online courses) often have a support 
infrastructure that is centered on the campus environment. Online students may have 
difficulty accessing library support, financial aid services, the technical help desk, 
advising and counseling, or tutoring. For example, Compora’s (2003) survey of six Ohio 
institutions found that none had a formal hotline or system to assist distance-education 
students with problems; to the extent that institutions had any system of assistance, it was 
either informal or passive (e.g., student-to-student e-mail discussion, faculty assistance, 
or instructions on the syllabus). In a study of online remedial math courses at a Florida 
community college, Zavarella (2008) found that students had access to tutoring only at 
the on-campus math lab; moreover, office hours were primarily offered on campus. 
Unsurprisingly, both services were poorly utilized by online students. As one student who 
withdrew from the online math class complained, “I could not get the support/help I 
needed!” (Zavarella, 2008, p. 85). Finally, while almost all campuses provide some type 
of technical support service for students, such services are not always open at the hours 
when students may be working online: A recent survey estimates that only 33% of two- 
and four-year institutions offer round-the-clock technical support (Green, 2010). 

When institutions do not explicitly provide non-instructional supports for online 
students, instructors are forced to take on these support roles (Compora, 2003; Kinser, 
2003), which is problematic for two interrelated reasons. First, instructors are not trained 
to be librarians, counselors, or technical support providers. Thus they are unlikely to 
provide students with the detailed, accurate, and timely responses that high-quality 
support services in these areas should provide. Second, to the extent that instructors 
attempt to provide these non-instructional services, they will necessarily spend less time 
on critical instructional responsibilities. 
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3.5 Online Completion Rates in Context 

The conclusion that online courses have higher withdrawal rates than face-to-face 
courses does not imply that face-to-face courses are an ideal standard for comparison. 
Community college administrators estimate face-to-face course completion rates to be 
approximately 76% (Instructional Technology Council, 2010), which certainly leaves 
room for improvement. In his 1999 book, Honored but Invisible, Grubb pointed out that 
community college instructors are often instructionally isolated, with few resources and 
supports to help them systematically improve their teaching. More recent research 
suggests that this situation has changed little over the past decade (Grubb, 2010). 
Moreover, as detailed by other working papers in this series (particularly Scott-Clayton’s 
[2011] paper on institutional complexity and Karp’s [2011] paper on non-instructional 
support), institutions have much more work to do to support underprepared students — 
whether in online or in face-to-face courses — to succeed. Section 5.2 of the paper makes 
several recommendations to help institutions improve online completion rates. It is worth 
nothing that many of these recommendations apply to face-to-face coursework as well. 



4. Low-Income and Underprepared Student Access 

A primary assumption underpinning the increase in online course offerings is that 
they increase educational access (Allen & Seaman, 2010; Cox, 2005; Epper & Gam, 
2003; Kuenzi, Skinner, & Smole, 2005; Parsad & Lewis, 2008). And indeed, there is no 
question that online learning improves access to college coursework on at least two 
fronts. First, in response to nearly every researcher who asks, students cite flexibility and 
convenience as primary advantages of online learning. These features make online 
courses particularly valuable to adults with multiple responsibilities and highly scheduled 
lives; thus, online learning can be a boon to workforce development, helping busy adults 
to return to school and complete additional education that otherwise could not fit into 
their daily routines. Second, online or hybrid modalities can allow colleges to offer 
additional courses or course sections to their students. A lack of available seats in key 
introductory courses can pose a barrier to student progression; freedom from the 
constraint of physical classroom space allows administrators to create as many course 
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sections as they can find qualified instructors for, which may lower the availability 
barrier. In addition, small colleges do not have the resources to offer certain highly- 
specialized courses to their students; an online partnership among institutions can allow 
these students to enroll in such courses via a partnering college. Finally, some colleges 
have leveraged the flexibility of online learning to redesign the nature of postsecondary 
education. Western Governors University’s entirely-online competency-based programs 
provide a personalized and flexible education, allowing students to forgo formal courses 
and potentially accelerate their completion of a degree. The Indiana Wesleyan University 
College of Adult and Professional Studies uses a cohort model, in which “weekly online 
classes are organized to begin when cohorts fill” (Auguste, Cota, Jayaram, & Laboissiere, 
2010, p. 13), a strategy that is thought to contribute to the college’s relatively high 
graduation rate. 

Given the convenience and flexibility of online learning, there is little doubt that it 
has induced many adults to return to school, particularly among the well-prepared and 
relatively high-income adult population for whom online learning was originally 
designed and implemented in continuing education departments across the country. Yet it 
is unclear whether online learning has induced new college enrollments among the low- 
income population. This review found no studies focusing on low-income or 
underprepared populations and whether their postsecondary enrollment has expanded as a 
result of the past decade’s explosion in online learning. Jaggars and Xu’s (2010) analysis 
of online course data across 23 colleges in the Virginia Community College System, 
however, suggests that the availability of online courses does not necessarily induce new 
enrollment. As part of a larger analysis, my colleague and I first examined the course- 
taking behaviors of students who initially enrolled in the fall of 2004 (see Figure 2). That 
fall, 13% of these students enrolled in online courses, but only 2% took all their credits 
online. Over time, students who persisted in school were increasingly more likely to take 
online courses; by the spring of 2008, 34% of the remaining students took at least one 
course online, and 13% took all their credits online. Next, we examined students who 
initially enrolled in fall 2008, expecting their online course-taking to be at a similar or 
higher rate than was the 2004 cohort’s spring 2008 numbers. Instead, we found that only 
17% took any course online and only 3% took all their credits online. The low online 
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course-taking rates among new students suggest that online coursework was not a strong 
inducement for new students to enroll in college. Rather, ongoing students were more 
likely to take advantage of the flexibility of online learning. 
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Figure 2 

Virginia Community College Online Enrollments, 2004 and 2008 Student Cohorts 



2004 Cohort 



2008 Cohort 





Key: 



Any online enrollment 
All online enrolment 



Note. Calculated from Tables 3.1, 3.3, 6.7, and 6.9 of Jaggars & Xu (2010). 
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It may seem intuitive that online learning would encourage new college 
enrollments among low-income populations, who could take advantage of its flexibility 
to juggle school, work, and family responsibilities. And yet the realities of college access 
can be counterintuitive; for example, consider the case of Pell grants. Studies of the Pell 
grant program (Hansen, 1983; Kane, 1996) found that it had no impact on new college 
enrollments among low-income populations. As Scott-Clayton (2011) discusses in her 
paper in this series, financial aid researchers suspect that the complexity of the Pell 
eligibility and application process obscures its benefits and prevents the program from 
reaching the individuals who need it most. Similarly, several factors may discourage low- 
income young adults from leveraging the flexibility of online coursework as an entry 
point into college. First, in order to make an investment in college, some students may 
seek online degree plans rather than online courses, and the supply of online degrees is 
limited. Second, technological infrastructure may pose a significant barrier to low-income 
students. And third, online courses typically do not substantially lower cost barriers to 
college. 

4.1 Online Courses Versus Online Degrees 

Most institutions feel that making more courses available is an important reason 
to offer online learning. However, fewer feel that making more degree programs 
available (55%) or certificate programs available (34%) is an important factor, and only a 
third of institutions offer at least one degree program entirely online (Parsad & Lewis, 
2008). Moreover, the range of degrees offered online is rather limited. For example, 
Western Governors University offers degrees in only four areas: Teacher Education, 
Business, IT, and Health Professions; and the range of majors within each degree is 
narrow. The University of Phoenix also offers a fairly narrow selection of online degrees. 

There may be a variety of reasons why institutions feel that offering a wide range 
of online degrees is not an important strategy, but one of the strongest reasons may be 
cost-effectiveness. It is far more cost-effective to design and deploy a single course with 
thousands of student enrollments per year (although these may be divided among many 
different instructors and sections) than to design a course with a few student enrollments 
per year (Jung, 2003). Rumble (2003) observes that: 



28 




For any given student population, the more courses on 
offer, the lower the average course population. However, 
students will rarely if ever be distributed equally across the 
courses. It is much more likely that in 80:20 rule will apply, 
with 80% of the students enrolled on something like 20% 
of the courses, so that one can expect a few courses have 
very high student populations and a large number to have 
relatively few students in them. Planners thus need to 
consider the likely student population on each course and 
bear this in mind in selecting the media to be used on each 
course, (p. 711) 

That is, while large-enrollment courses may be deployed more cheaply online 
(although that is not a foregone conclusion; see Inglis, 2003; Jung, 2003; Rumble, 2003), 
small -enrollment courses may be deployed more cheaply face-to-face. Given that most 
degrees require a mixture of large-enrollment introductory courses and small-enrollment 
advanced courses, offering an entire degree plan online may be cost effective only if: (1) 
the institution has a wide geographical reach and large base of potential enrollees for the 
degree, as the University of Phoenix Online does; and (2) the institution offers a fairly 
limited range of highly-popular degrees online, such that even relatively small courses 
have large aggregate enrollments each year. 

4.2 At-Home Technology Barriers 

While the “digital divide” has perhaps disappeared in terms of overall Internet 
use, low-income households are still at a significant disadvantage in terms of the 
relatively sophisticated infrastructure required to participate in online learning. In 1999, 
the cost of an up-to-date computer was still a problem for 20% of community college 
students (Phillippe & Valiga, 2000). 6 In 2009, only 42% of households with incomes less 
than $30,000, 46% of adults who had at most a high school degree, 52% of African- 
Americans, and 47% of Hispanics had high-speed Internet access at home (Rainie, 2010). 
These numbers have not grown substantially since 2006 (cf. Rainie, Estabrook, & Witt, 
2007). This may be because low-income groups, particularly African Americans, 



6 Although the cost of a computer has decreased in the last decade (Bureau of Labor Statistics, 2010), there 
is still reason to believe that many low-income students cannot afford an up-to-date computer: in a recent 
study (Fairlie & London, 2009), the simple treatment of providing at-home computers to community 
college students who were on financial aid resulted in increased academic success. 
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disproportionately access the Internet via handheld devices rather than through at-home 
broadband (Horrigan, 2009). 

Students with outdated at-home computers or dial-up Internet access will 
experience agonizing delays when attempting to interact with most course delivery 
platforms. Those with insufficient infrastructure at home might go to a friend’s house, an 
Internet cafe, the library, 7 or to a local satellite campus to work on an online course. 
(Anecdotally, administrators have reported to CCRC researchers that they see students 
drive to the college parking lot, where wireless is available, to do coursework with their 
laptop in the car.) Given the juggling of schedules, transportation, and costs associated 
with these various options, an online course may be no more convenient to these students 
than would be an evening or weekend course at the local community college. Indeed, one 
study in the Dallas Community College District suggests that night and weekend classes 
are preferred over distance classes by Hispanics, which represent a large proportion of the 
district’s population (Borcoman, 2004). 

4.3 Other Cost Barriers 

When asked to choose from a list the option that would most help them return to 
school (Johnson & Rochkind, 2009), only 7% of college dropouts said that putting 
classes online would help the most — far below the proportion of those who selected 
cutting the cost of college by a quarter (25%), providing more loans (14%), or allowing 
part-time students to qualify for financial aid (13%). While online learning can help 
ameliorate access barriers in terms of scheduling and location and may reduce some 
transportation and child care costs, it does not reduce educational tuition and fees. These 
costs are a particularly substantial barrier for lower-income working adults, who often 
cannot qualify for financial aid (Chao, DeRocco, & Flynn, 2007). 

Proponents of online learning have argued that it is more cost effective to 
institutions, and presumably those cost savings should trickle down to online students. 
However, it is far from established that offering online sections saves money to 
institutions (Campus Computing Project, 2009; Compora, 2003; Rumble, 2003), and 
some argue that online courses can only be designed and distributed more cheaply than 

7 Public library closures and budget cuts across the country are increasingly limiting the option of accessing 
the Internet for free at the local library (e.g., Brodsky, 2010). 
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face-to-face courses if their quality is sacrificed (Hanna, 2003; Maeroff, 2003). Even if 
online course provision is more cost effective for institutions, those savings are not 
currently passed to students; rather, 40% of community colleges charge students 
additional fees for online education (Instructional Technology Council, 2010). 



5. Recommendations to Improve Access and Progression 

5.1 Improving Low-Income Student Access 

The following recommendations may help online learning fulfill the promise of 
expansion of access to postsecondary education for low-income and underprepared 
students. 

Reduce direct costs to low-income students. In order for expanded online 
learning to translate to increased access, the cost to students must be dramatically 
reduced, both in terms of tuition and at-home technological infrastructure. For example, 
to implement an online workforce development program for low-income single mothers, 
program administrators provided students with an at-home computer, printer, Internet 
access, and skills training at no cost for a year (Gatta, 2005). To reduce student costs on a 
larger scale, initiatives to expand education through freely-available online courses may 

o 

be a step in the right direction, particularly if paired with a low-cost provision of high- 
speed Internet access and laptops to low-income students (for example, see Fairlie & 
Fondon, 2009). However, it is not yet clear how quality instruction and effective student 
support services, discussed in more detail below, could be maintained in a tuition-free 
environment. In order to reduce students’ out-of-pocket technology costs, some colleges 
have also experimented with techniques to cover these costs through financial aid. For 
example, if a high-speed Internet card is a course requirement, students may be able to 
purchase it at their campus bookstore using financial aid. 



8 For example, the Next Generation Learning Challenge, a collaborative partnership among Educause, the 
League for Innovation in the Community College, the International Association for K-12 Online Learning, 
and the Council of Chief State School Officers with funding from the Bill & Melinda Gates Foundation, is 
supporting the development of open courseware for a core set of common postsecondary courses. More 
information is available at http://nextgenleaming.com/the-challenges/open-core-courseware 



31 





Revise financial aid structures. Costs to students can also be decreased through 
changes to the financial aid structure. Chao, DeRocco, and Flynn (2007) recommend that 
Pell grant restrictions be loosened to allow year-round college attendance, no reduction in 
aid for attendees of low-tuition community colleges, and distance education programs 
leading to one-year or shorter certificate programs. Some states also place restrictions on 
financial aid for students in online degree programs, and perhaps it is time for these rules 
to be re-examined. Perhaps even more importantly, these potential cost decreases should 
be part of a legislative package that clarifies and simplifies eligibility rules, in order to 
encourage more low-income students to apply for grants in the first place (Dynarski & 
Scott-Clayton, 2006). 

Create more fully-online programs. If community colleges and other 
institutions that predominantly serve low-income and underprepared students wish to 
draw new enrollees via online coursework, they may need to consider how to design and 
fund fully-online degree programs. For some institutions, cost-effective provision of an 
entire degree program may require a partnership with other schools or organizations that 
provide online learning. It is also imperative to gain faculty support, and in general, 
faculty will be hesitant to support a new online program unless they are confident of its 
quality and feel assured that students in the program have a chance of success equal to 
that of their face-to-face peers. That is, the institution’s ability to expand access may be 
in part dependent on whether it can first solve the challenges of online course completion 
and progression. 

5.2 Improving Online Completion and Progression 

This review of the literature suggests that community college students are much 
less likely to complete online courses than face-to-face courses and also may be less 
likely to progress to subsequent courses after taking an online course. To improve student 
completion and progression, it is recommended that colleges: 

Assess student ability to succeed. In order to reduce high online course 
withdrawal rates, many colleges recommend online courses only to students whom they 
feel are prepared to succeed in those courses (e.g., S. Y. Liu, Gomez, Khan, & Yen, 

2007; Millward, 2008). These assessments should be used to communicate the 
expectations of online coursework and to improve students’ self-awareness of how their 
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academic assets match (or do not match) the features and challenges of online learning. 
Similarly, some technology-related problems can be reduced by limiting enrollment to 
students who have the appropriate technology skills and at-home infrastructure. These 
measures, however, are not always effective, as they rely on student self-assessment 
(Millward, 2008). Moreover, they reduce online dropout simply by restricting the 
population of students who enroll in online courses, which could inadvertently undercut 
the institution’s access agenda. Accordingly, it is recommended that in addition to the use 
of assessments and advisories, colleges also concentrate on improving the success of all 
students who choose to enroll online, using assessments to help inform the programming 
and supports discussed below. 

Teach online learning skills. Colleges can assist students in building requisite 
skills prior to taking an online course, such as through a prerequisite computer literacy 
course (e.g., Erlich, Erlich-Philip, & Gal-Ezer, 2005). Such policies, however, have two 
potential drawbacks. First, if a computer literacy or similar course is merely 
recommended and not required, some students will ignore that recommendation. Second, 
if the recommendation is enforced as a requirement, it may inadvertently undercut access 
to online courses by discouraging students who feel they cannot spend time or money on 
an extra course. In order to support both access to and success in online courses, 
institutions could either: (1) provide incentives to students to build their skills prior to 
online course enrollment, for example, by offering reduced fees in subsequent online 
courses; or (2) provide struggling students with the scaffolding and supports they need 
within the framework of entry-level online courses. 

Enhance non-instructional supports. To remove the burden of non-instructional 
support from the shoulders of instructors and improve the level of supports offered to 
students, colleges should more seriously consider how to provide high-quality and easy- 
to-access online learning supports. For example, Western Governors University online 
students work with a dedicated mentor throughout their college program, following an 
individually-tailored Academic Action Plan. Similarly, at the online community college 
Ivy Bridge College, each student is assigned a coach, who checks on the student’s 
progress on a weekly basis (Moltz, 2011). University of Phoenix Online provides its 
online students with an online library, a math resource center, a writing resource center, 
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and a “graduation team” consisting of a finance advisor, enrollment advisor, and 
academic advisor. 9 

Most institutions cannot afford to provide online students with individual coaches 
who will provide guidance on a regular basis. And at first blush, it may also seem 
prohibitively expensive for institutions to provide a full range of round-the-clock support 
services to its online students. However, in their respective papers in this series, Scott- 
Clayton (2011) and Karp (2011) discuss promising solutions to this quandary. Applying 
their discussions to this particular issue, I would make three suggestions. 

First, in order for students to use support services, they must be seamlessly 
integrated into the spaces in which students already live and work (Neal & Jaggars, 

2010). At a minimum, links to traditional on-campus support services should be 
displayed prominently on the course’s web interface, with information about how 
students can access these services via e-mail, chat, or telephone. Students will be even 
more likely to take advantage of support services if they are explicitly incorporated into 
class activities. For example, in an online introductory history class, the instructor could 
require students to consult with the writing center on the course’s first writing 
assignment. To make this vision a reality, campus administrators would need to 
orchestrate collaboration between support providers and academic departments, with the 
aim of creating a set of support-oriented academic activities that would be systematically 
built into the curricula of the most common introductory courses taught online. 

Second, with assistance from practitioners and researchers who can help identify 
the most important components and characteristics of non-academic support, stakeholders 
in the field should design and test automated systems that could dynamically provide 
those key components, without the need for round-the-clock human staffing. Unusual 
problems could then be handled by well-trained support staff during regular working 
hours. 

Third, to cost-effectively expand service availability (whether through an 
extension of traditional service hours or through the provision of a sophisticated dynamic 



9 Of course, such supports are not inexpensive. Like most private for-profit institutions (Bailey, 2006), the 
University of Phoenix charges substantially higher tuition than public two-year institutions, and it also 
charges higher tuition for online than on-ground courses, presumably to maintain their wrap-around support 
services (University of Phoenix, n.d). 
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online system), colleges should consider partnering with other institutions or 
organizations to capitalize on economies of scale, providing a single set of support 
systems to multiple campuses. Of course, this option is feasible only if the set of 
partnering institutions is willing to agree on a consistent set of systems; for example, 
campuses would need to share the same web-based platform (for technical support), 
course numbering and program requirements (for course selection support), course 
curricula (for tutoring support), and information resource systems (for library support), 
and would also need to share policies in certain areas (for example, guidelines for 
appropriate writing-center support). Accordingly, the shared- support strategy may be 
most feasible within large districts or state systems wherein all colleges share similar 
infrastructures and policies. Creating consistent infrastructure and supports across a state 
or district would also reduce confusion and support needs among students who take 
courses from multiple colleges. 

Enhance instructional supports. To overcome the social distance inherent in 
distance learning, engaging students and motivating them to succeed in the class, it is 
recommended that colleges: 

Intentionally design online courses. To create consistently high online retention 
rates, instructors may need to spend substantial amounts of time designing and 
implementing “scaffolding” activities, moderating discussion, and encouraging struggling 
students. As face-to-face courses are translated into online courses, these elements need 
to be explicitly built into the course. The support of instructional technology 
professionals is key in building effective online courses; however, course design should 
not be relegated entirely to individuals who will not be teaching the course. Koehler and 
colleagues (Koehler, Mishra, Hershey, & Peruski, 2004) argue persuasively that in order 
to appropriately marry content, technology, and pedagogy, instructors need to be integral 
to course design. Moreover, course designers should be aware that simply adding 
technical bells and whistles (such as embedded video or online quizzes) to online courses 
is unlikely to result in improved student outcomes (U.S. Department of Education, 2009). 
Such tools will be effective only to the extent that they promote mechanisms that may be 
important to student retention and learning, such as enriching interaction (Bernard et al., 
2009), helping students reflect on their own learning (U.S. Department of Education, 
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2009), or improving system knowledge and self-efficacy (Hannafin et al., 2003). For 
example, as Grubb (2010) points out, many computer-based tutoring programs are based 
on “the dismal practices of remedial pedagogy” (p. 28); before incorporating computer- 
based tutoring into an online course, then, course designers need to carefully evaluate the 
quality of the program’s pedagogical design. 10 

Support faculty development. To provide high-quality curricula and instruction, 
faculty need strong support from the institution. Well-regarded online courses are often 
designed through a team-based approach, with faculty collaborating with an instructional 
designer, and often with additional support staff (e.g., Alvarez, Blair, Monske, & Wolf, 
2005; Hawkes & Coldeway, 2002; Hixon, 2008; Knowles & Kalata, 2007; Puzziferro & 
Shelton, 2008; Thille, 2008; Xu & Morris, 2007). Yet CCRC’s national field study of 15 
community colleges (Cox, 2006) found that most faculty were left to design online 
courses on their own and that training for online instructors was primarily focused on 
technical aspects of the online course management system. None of the colleges offered 
faculty the degree of expert support they needed to redesign curricula and pedagogical 
strategies for the online context. Worse, some institutions had policies that actively 
undercut faculty engagement in online learning, such as not counting online courses as 
part of a normal teaching load, or enrolling twice the number of students in online as 
compared to face-to-face sections. As a result, one college’s online coordinator said: 

Probably the same range of practices that we have in our 
face-to-face courses is being replicated online, when that 
might have been an opportunity not just to teach people 
technology, but to have them think about engaging students 
and what kinds of things you can do better online, rather 
than just post your lecture, or whatever it is that might be 
happening, (p. 125) 

Similarly, a recent survey of community college English teachers (Millward, 
2008) found that while most colleges offer training to online instructors, faculty were 
dissatisfied with the training programs’ focus on technology rather than pedagogy. 
Moreover, 43% indicated that they received no compensation for the time they spent on 
training. Overall, the survey found, as Millward summarizes, that online instructors “are 

10 For more on pedagogical elements that may improve student learning, see working papers in this series 
by Hodara (2011) and Perin (2011). 
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asking for more access, more training, more time for innovation and implementation, and 
more research.” 

Koehler et al.’s (2004) study illustrates a relatively cost-effective way to engage 
instructors in online course development: Faculty enrolled in a semester-long educational 
technology course, in which they designed a new online course in concert with two 
graduate students. The faculty received a new laptop and a $1,000 stipend for course 
development, but received no course buyout or overload pay. Anecdotally, some 
community colleges require faculty to participate in intensive online course development 
and pedagogical training before they are considered qualified to teach online. These 
approaches seem promising, although as yet there is no solid evidence as to their 
effectiveness. 

Engage in continuous improvement efforts. As Jenkins (2011) discusses in his 
paper in this series, an institution is unlikely to substantially improve student success 
unless it engages in a systematic long-term improvement process. It may not always be 
clear exactly which instructional designs, practices, and supports will improve online 
student success at a given institution. However, institutions can begin to identify and 
refine successful practices through a process of continuous quality improvement. Dietz- 
Uhler and colleagues (Dietz-Uhler, Fisher, & Han, 2007-2008) provide one example of 
how to begin such a process using the well-known Quality Matters framework. Like other 
online quality frameworks (see Parker, 2004), Quality Matters offers a set of general 
guidelines and principles, but does not specify exactly what institutions or instructors 
should do to meet those guidelines. Through a careful approach of peer review, outcome 
measurement, and subsequent adjustments, departments and colleges can begin to 
develop specific practices to meet quality guidelines and improve student outcomes. In 
this process, institutions may wish to supplement general quality guidelines with 
measurements that are more specific to instruction and online interaction. For example, 
the American Public University System engages in continuous quality improvement of its 
online courses using a course-based student survey built on the Community of Inquiry 
framework (see Boston et al., 2009; Arbaugh et al., 2008; Ice, 2009). Using survey 
outcomes, the system identifies programs and instructors with significantly higher scores 
and examines their work for innovative or exemplary practices that could potentially be 
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implemented by lower-scoring programs and instructors. Perhaps most importantly, 
program directors review individual faculty outcomes with each instructor, constructively 
discuss potential ways to improve scores, and incorporate these reports into quarterly 
audits. Finally, in addition to assessing quality through peer review and student surveys, 
it will be most helpful for departments to set ambitious standards for course learning 
outcomes and to continuously assess and improve the extent to which students meet these 
outcomes (Jenkins, 2011). 

Of course, a quality improvement process should not be limited to online courses 
alone. Face-to-face courses would also benefit from continuous assessment and revision. 
Online program administrators, however, may have an advantage in terms of pushing a 
continuous improvement agenda for two reasons. First, online learning is relatively new 
to many faculty, and their experience with it is still evolving. Moreover, according to a 
recent survey (Green, 2010), a vast majority of colleges have either recently completed or 
expect to soon implement a reorganization of online learning. It may be more politically 
viable to introduce a new quality improvement approach within the still-shifting context 
of online learning than within the relatively traditional context of face-to-face learning. 
And second, online course management systems offer the possibility of far more 
advanced learner analytics than is possible in face-to-face learning, and these 
sophisticated data might feed more readily into a continuous improvement approach. For 
example, the Open Learning Initiative captures transactional data on all student learning 
activities and uses the resulting data to revise each course for the following semester 
(Thille, 2008). 

Conduct further research. In order to provide institutions and individual faculty 
with more specific research-based guidance on how to improve online success rates, 
foundations and governmental organizations should also: 

Fund research to clearly define key elements of quality instruction. Despite 
suggestive themes from theory and research, the specific characteristics and 
measurements of “high-quality curricula and pedagogy” remain elusive. The research 
literature lacks a consistent set of measurements and definitions for assessing curricular 
and pedagogical elements of online coursework (see Kirschner, Strijbos, Kreijns, & 

Beers, 2004; Strijbos, Martens, Prins, & Jochems, 2006). As a result, studies tend to 
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focus on elements that are easy to measure. For example, many studies of online 
interaction focus on frequency of interaction, rather than on the quality of that interaction 
(Hull & Saxon, 2009). Similarly, many studies compare learning outcomes between a 
group that used a specific technological tool meant to facilitate interaction versus a group 
that did not use it (i.e., a measurement of “tool versus no tool”), without measuring the 
patterns of instructor and student behaviors evoked by the tool (for a review, see Bernard 
et al., 2009). If researchers do not have consistent definitions for elements of interaction 
and other key instructor and student behaviors across studies, they cannot explain why 
particular tools or strategies seem effective or ineffective, nor can they extrapolate to 
create guidelines for designing more effective instruction. 

Foster collaborative research activities. To move forward, practitioners and 
researchers need to work closely together. Practitioners can begin by developing online 
curricular and pedagogical interventions that are well grounded in theory. For example, 
Strijbos and colleagues (Strijbos, Marten, & Jochems, 2004) use theoretical principles of 
group communication to set forth a framework for designing computer-supported group- 
based learning activities. Their framework does not specify how a given activity should 
be structured, but instead provides a basis for clearly and consistently measuring the 
elements of that structure, as well as how the structure impacts changes in group 
interaction. Researchers can then subject these well-measured and clearly-understood 
elements to rigorous empirical testing in terms of their impact on student retention and 
learning. In addition, qualitative research that examines not only online student 
engagement and motivation (e.g., Bambara et al., 2009) but also how specific curricular 
and pedagogical elements affect those psychological constructs may be particularly 
helpful in terms of guiding course design. 



6. Conclusion 

Online learning affords flexibility and convenience to students and also allows 
institutions to be more flexible in their approach to education, in both narrow ways (e.g., 
offering more sections of course than can be accommodated by the college’s physical 
classroom space) and very broad ways (e.g., enrolling new students each month into 
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competency-based programs, a la Western Governors University). Yet students also 
encounter challenges in online coursework as it is typically implemented — including 
technical difficulties, a sense of isolation, a relative lack of structure, and a general lack 
of support — that may contribute to low completion rates, particularly among community 
college students. To help ameliorate these problems while still allowing for increased 
flexibility, some educators advocate the expansion of hybrid coursework, which is 
thought to provide students with the “best of both worlds.” For example, most courses 
redesigned under the National Center for Academic Transformation process are modified 
to become hybrid courses or technology-assisted face-to-face courses, rather than to 
become online courses (Twigg, 2003). Unlike online courses, however, hybrid courses do 
not offer complete freedom from geographic and temporal constraint, and thus do not 
hold out the same promise for dramatically improved access to postsecondary education. 
Accordingly, online learning has an important role in community college education — a 
role that will likely continue to grow in scope and consequence. 

In order for online learning to meet its promise of improving postsecondary 
access and success among low-income and underprepared students, this paper 
recommends a two-pronged strategy of practical investments and research investments. 

In terms of policy and practice, this paper recommends reducing the cost of online 
learning by offering low-cost computers and high-speed Internet to aspiring students; 
restructuring financial aid to allow students to pay for the remaining costs of enrollment; 
providing integrated and high-quality non-instructional supports into the online course 
management system; providing online instructors with the resources and training they 
need to implement high-quality online curricula and pedagogies; and implementing peer- 
review-driven continuous quality improvement processes. 

In terms of research, we need more information on how online learning can 
improve low-income student access, as well as the extent to which new policies and 
practices to reduce the cost of online education could further improve access. Further, 
researchers need to work to isolate the key elements and mechanisms of effective non- 
instructional supports, and to identify the instructional behaviors and activities that 
encourage student engagement, motivation, retention, and learning. To accomplish this 
end, foundations and governmental research organizations will need to strongly promote 
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and support the creation of theoretically-driven frameworks and research designs that 
make use of clear and consistent measures of student and instructor behavior and that 
explicitly link those patterns to student academic outcomes. 

Many of these recommendations will require a substantial investment of new 
resources into online learning. There is still very little concrete information as to the real 
cost-effectiveness of online coursework under the current set of practices; there is also 
very little information on whether online learning can provide savings to institutions 
without compromising student success. Further research in this important area is certainly 
needed. It is clear, however, that in order improve the output of online courses, it is 
necessary to improve the inputs into the system, which may threaten current cost models 
for online education. Thus, any substantial improvements to the effectiveness of online 
learning may require new cost models, designed in collaboration among educational 
foundations, state and federal government, and college systems. 
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Appendix 



Table 1 

Summary of Studies 



First Author 
(Year) 


Study Design 


Controls 


Online Course Characteristics 
& Sample Size 


Outcomes 

Examined 


Summary of Findings 


Blackner (2000) 


Comparison using 
controls (regression) 


Math anxiety, locus of control, 
learning style 


Community college 
developmental math courses 
(N = 42) a 


Completion, 
exam score, 
course grade 


Grades similar but online withdrawal 
rates higher 


Block (2008) 


Comparison of gain 
scores 


None 


University health & wellness 

course 

[N = 19) h 


Exam score 


Gain scores similar 


Brown (2002) 


Comparison using 
controls (regression) 


Female, Black, math coursetaking, 
ACT score, GPA, previous credits 


University microeconomics 

course 

(N = 89) g 


Exam score 


Online exam scores lower 


Caldwell (2006) 


Randomized 


None 


Historically Black university, 
introductory C++ course 
( N = 20) b 


Assignment 
& exam 
scores 


Assignment & exam scores were 
similar 


Carey (2001) 


Comparison of gain 
scores 


None 


University management 
information systems course 
[N = 103) a ' b 


Exam score 


Gain scores similar (did not take into 
account observed higher online 
withdrawal rates) 


Carpenter (2004) 


Comparison using 
controls (regression) 


Gender, ethnicity, registration 
timing (late vs. early), credit load, 
age, placement exam scores 


Community college 
developmental writing course 
(N = 256) x 


Completion, 
Course grade 


Online withdrawal rates higher; but 
among those who completed, online 
grades higher 


Coates (2004) 


Comparison using 
controls (regression) 


30 controls (e.g., SAT, hours 
employed, commuting time to 
campus, age, pre-test, major) 


University economics, 3 
courses 

( N - 59) b * c ° urse 2 *' d l c ° urse !) 


Exam score 


Online exam scores lower 


Dutton (2001) 


Comparison using 
controls (regression) 


Undergraduate status, homework 
grades 


University C++ course 
(N = 141) e ' h 


Completion, 
exam score, 
course grade 


Completion similar; online exam 
scores & course grades higher 


Dutton (2002) 


Comparison using 
controls (regression) 


Homework grade, undergraduate 
status, experience with computing, 
employment, age, commuting time, 
has childcare responsibilities, 
female, hours enrolled 


University C++ course 
(N = 89) e,h 


Completion, 
exam score, 
course grade 


Completion, course grades similar; 
online exam scores higher 


Dutton (2005) 


Comparison using 
controls (regression) 


Prior GPA, homework grade, 
commuting time, computer 
experience, female (learning 
outcomes); prior GPA (completion) 


University business statistics 

course 

(. N = 48) a ' b 


Completion, 
exam score, 
course grade 


Completion similar; online exam 
scores & course grades higher 
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First Author 
(Year) 


Study Design 


Controls 


Online Course Characteristics 
& Sample Size 


Outcomes 

Examined 


Summary of Findings 


Ferguson (2009) 


Comparison using 
controls (ANCOVA) 


GRE scores 


Graduate education course 
[N = 44) x 


Assignment 
scores, exam 
score 


First assignment scores similar; 
online second assignment scores 
lower; online final exam scores 
lower 


Figlio (2010) 


Randomized (from a 
pool of volunteers) 


Ethnicity, gender, prior GPA (one at 
a time) 


University microeconomics 

course 

(N = 215) g 


Exam scores 


Exam scores similar overall, but 
online exam scores lower among 
Hispanics, males, and low-prior-GPA 
students 


Grayson (2001) 


Comparison using 
controls (ANCOVA) 


Prior grades, learning style 


University critical reasoning 

course 

[N = 35) b 


Participation 
scores, 
assignment & 
exam scores, 
course grade 


Online students' participation scores 
lower, online course grade 
marginally lower, other outcomes 
similar 


Herman (2004) 


Comparison using 
controls (ANCOVA) 


Prior GPA, pretest 


University computer basics 

course 

[N =18) h,f 


Exam score, 
course grade 


Similar exam scores (using both 
controls); online course grade higher 
(using only GPA control) 


Jaggars (2010)* 


Comparison using 
controls (multilevel 
regression) 


20 controls (e.g., age, female, 
ethnicity, ESL student, financial aid 
eligibility, credit load, prior GPA) 


Community college courses 
(2004 cohort 

N = 10,244; 2008 N = 7,665) x 


Completion, 

retention, 

transfer/grad 

uation 


Online completion rates, next- 
semester retention, and eventual 
graduation/senior-college transfer 
rates lower 


Mentzer (2007) 


Randomized (from a 
small pool of 
volunteers) 


None 


University education course 
(N= 18) c ' d 


Exam scores, 
course grade 


Similar exam scores; online course 
grades lower 


Musgrove (2002), 
study 1 


Comparison using 
controls (ANOVA) 


Learning style 


Community college nursing 

courses 

(N = 65) a 


Course grade 


Course grades similar 


Musgrove (2002), 
study 2 


Comparison using 
controls (ANOVA) 


Group embedded figures test 


Community college nursing 

courses 

(N = 53) a 


Course grade 


Course grades similar 


Navarro (2000) 


Comparison using 
controls (ANCOVA) 


Prior GPA, gender, ethnicity, class 
level, major (one at a time) 


University macroeconomics 
[N = 49) a,e 


Exam score 


Online exam scores higher 


Parkhurst (2008) 


Comparison of gain 
scores 


None 


University engineering 
[N = 84) c,g 


Exam score 


Online gain scores higher 


Partrich (2003) 


Comparison using 
controls (ANOVA) 


Learning style 


Graduate theology courses 
[N = 62) x 


Course grade 


Course grades similar 


Peterson (2004) 


Comparison using 
controls (ANOVA) 


Pretest 


Graduate education courses 
(N = 38) b ' d,i 


Project 

grades 


Project grades similar 
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First Author 
(Year) 


Study Design 


Controls 


Online Course Characteristics 
& Sample Size 


Outcomes 

Examined 


Summary of Findings 


Piccoli (2001) 


Randomized 


Gender, instructor 


University management 
information systems course 
(N = 94) a,f 


Exam scores 


Exam scores similar (did not take 
into account seemingly higher online 
withdrawal rates) 


Poirier (2004) 


Randomized (from a 
small pool of 
volunteers) 


None 


University psychology course 
[N = 12) b ' h 


Assignment 
& exam 
scores 


Online exam scores (4 exams over 
entire semester) higher; assignment 
scores similar 


Rosenfeld (2005) 


Comparison using 
controls (ANOVA) 


Course subject, age, race, gender, 
previous credits accrued (one at a 
time) 


Community college English, 
math, history courses 
(N = 402) 


Completion 
(with C or 
better) 


Online completion rates lower 


Ross (2000) 


Comparison using 
controls (ANCOVA) 


Pretest, cognitive learning style 


University 'law and disability' 

course 

[N = 36) g 


Exam score 


Online exam scores higher 


Sankaran (2001) 


Comparison within 
each level of each 
control 


Learning strategy, general 
motivation (one at a time) 


University accelerated 
(4-week) business computer 
course 
[N = 46) x 


Exam score 


Similar exam scores 


Scheines (2005), 
Study 1 


Randomized (from 
volunteers wanting 
online course) 


Pretest 


University critical reasoning 
course 

(N unclear, approx. 60, 
or 180/3) 9 ' h 


Exam scores, 

discussion 

attendance 


Similar exam scores (controlling for 
pretest); online discussion 
attendance lower (no control) 


Scheines (2005), 
Study 2 


Randomized (from 
volunteers wanting 
online course) 


Pretest 


University critical reasoning 
course 

(N unclear, approx 43, 
or 130/3) 9 ' h 


Exam scores, 

discussion 

attendance 


Similar exam scores (controlling for 
pretest); online discussion 
attendance lower (no control) 


Schoenfeld- 
Tacher (2001) 


Comparison using 
controls (ANCOVA) 


Pretest 


University science course 
(N = ll) e 


Exam scores 


Online exam scores higher 


Summerlin (2003) 


Comparison using 
controls (regression) 


Reading ability, age, gender, 
ethnicity 


Community college 
developmental math course 
(N = 79) a,f 


Exam score 


Similar exam score (did not take into 
account higher online withdrawal 
rates) 


Vroedingay 

(2005) 


Comparison using 
controls (ANOVA) 


Gender, marital status, child & 
employment information, highest 
education level, proficiency scores 
(one at a time) 


Private for-profit college 
environmental science course 
(N = 80) bh 


Exam score, 
course grade 


Similar exam score, similar course 
grades 


Wang (2004) 


Comparison using 
controls (ANOVA) 


Pretest, learning style 


Graduate education courses 
[N = 31) x 


Exam score 


Similar exam scores 


Waschull (2001), 
Study 2 


Randomized 


None 


Community college psychology 

course 

(N = 18) h 


Completion, 
exam scores 


Completion, exam scores similar 
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First Author 
(Year) 


Study Design 


Controls 


Online Course Characteristics 
& Sample Size 


Outcomes 

Examined 


Summary of Findings 


Xu (2010) 


Comparison using 
controls (multilevel 
regression/PSM) 


22 controls (e.g., age, female, 
ethnicity, ESL student, financial aid 
eligibility, credit load, prior GPA) 


Community college 
'gatekeeper' English course 
(N = l,052) x 


Completion, 
course grade 


Online completion rates and course 
grades lower 


Zavarella (2008) 


Comparison using 
controls (regression) 


Age, gender, ethnicity, marital 
status, learning style, reason for 
format choice, placement exam 
score 


Community college 
developmental math course 
[N = 56) a,f 


Completion 


Online completion rates lower 



Notes. "Controls" refers only to those used when analyzing the outcomes examined in this table. Stratification variables included in the analysis of an outcome are 
considered equivalent to controls. "Sample size" refers to the N of the online group. Some studies analyzed more outcomes than those listed here. The table includes 
access, completion, performance, and progression outcomes that were compared either between randomized groups, or between groups taking into account at least 
one control variable. Comparison of gain scores from pre- to post-test is considered equivalent to a comparison of post-test while controlling for pre-test. Each 
superscript a though / is omitted if it was not explicitly noted in the study's description of the online course(s). For example, if a study's authors note that asynchronous 
discussion was provided as part of the online course platform, but do not explicitly note that participation was a course requirement, superscript b is omitted. 

*Jaggars & Xu (2010). 

a Initial face-to-face session required. Asynchronous discussion participation required. Synchronous discussion participation required. Student collaboration/ group 
work required. '"Virtual laboratory" incorporated in online courseware. ^Tutorial software was primary teaching tool. 6 Video/audio of lectures was primary teaching 
tool. h Text-based or slideshow-based material was primary teaching tool (some supplemented with video/audio clips). 'Course explicitly designed to meet specific 
quality standards. x No relevant information provided. 
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